Appl. No. 09/904,253 

Remarks 

This is in response to the Office Action dated June 22, 2005. The Office Action first 
objected to claims 2, 6, 1 1, 12, 16, 19 and 29 for various informalities. The Office Action next 
rejected claims 1-53 under 35 U.S.C. §101 as being directed to nonstatutory subject matter. 
Finally, the Office Action also rejected claims 1-53 under 35 U.S.C. § 102(b) as being anticipated 
by the Sharma et al. publication ("Toward Multimodal Human-Computer Interface", Proceedings 
of the IEEE, vol. 86, Issue 5, May 1998, pages 853-869). 

Applicants have amended claims 2, 6, 7, 1 1, 12, 19 and 29 to correct the informalities 
identified by the Office Action. Claims 1 , 2, 3 and 45 have been amended in response to the 
rejection under 35 U.S.C. §101. Applicants traverse the rejections under 35 U.S.C. § 102(b). 

Claims 1-53 remain under consideration. 

Objections to the Claims: 

The Office Action first objected to claims 2, 6, 1 1, 12, 16, 19 and 29 for various 
informalities. In response to these objections, applicants have amended claims 2, 6, 7, 1 1, 12, 19 
and 29. Applicants could not determine how the objections raised were intended to be applied to 
claim 16. If the Office persists in this objection to claim 16, applicants request that the details of 
that objection be pointed out with particularity. 

Regarding claim 2, the Office Action indicated that it was not clear what the Applicant 
meant by the term "that". In response, Applicants have deleted this term for clarity. 

Regarding claim 6, the Office Action indicated that there may have been some claim 
language missing. In response, Applicants have amended claim 6 for clarity to read: 

"The multimodal recognition system of claim 5, wherein the multimodal recognition 
subsystem further comprises a third subsystem that inputs the second finite-state 
transducer and outputs a recognition based on said at least one finite-state machine. 

Thus, claim 6 is now clear. 

Claim 7 has been amended to conform to the amendment of claim 6 to now claim, in part: 

a multimodal recognizer that inputs said at least one finite-state machine and outputs the 
multimodal recognition based on said at least one finite-state machine and the third finite- 
state transducer. 
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Thus, claim 7 is also clear and possesses adequate antecedent basis. 

Regarding claim 1 1, the Office Action indicated that it was not clear what was meant by 
the term "that other mode recognition system." In response, Applicants have amended claim 1 1 
to state, in part: "at least one other mode recognition subsystem in said plurality is usable as a 
recognition model by said at least one other mode recognition subsystem." Accordingly, claim 
1 1 is now clear regarding the reference to the at least one other mode recognition system in said 
plurality. 

Regarding claim 12, Applicants presume that the Office Action objected to the terms 
"each other mode recognition system" and "that other mode recognition system" as originally 
claimed in claim 12. In response to this presumed objection, applicants have amended claim 12 
to replace each of these phrases with the phrase "said at least one other mode recognition 
subsystem." Thus, claim 12 is now clear and possesses adequate antecedent basis for all terms. 

Regarding claim 19, the Office Action stated it was not clear how the second mode 
related to the first mode to a meaning of a combination of the first and second modes. Also, the 
Office Action objected to the phrase "a possible meaning." In response, Applicants have 
amended the first element of claim 19 to claim: 

the third finite-state transducer relates the first one of said plurality of modes and the 
second one of said plurality of modes to a meaning of a combination of said first one of 
said plurality of modes and said second one of said plurality of modes; . . . 

Thus, it is now clear that the third finite-state transducer functions to relate the first one of said 
plurality of modes and the second one of said plurality of modes to a meaning of the combination 
of those modes, as identified by the recognition system. Applicants have amended the second 
element of claim 19 to delete the word "possible" and, therefore, the second element of claim 19 
now clearly claim that the "meaning subsystem . . . outputs ... a meaning lattice based on the 
first finite-state machine and the third finite-state transducer." Thus, claim 19 is now clear and 
possesses adequate antecedent basis for all terms. 

Finally, regarding claim 29, Applicants presume that the Office Action intended to object 
to this claim because the terms "the gesture mode" and "the speech mode" did not contain 
adequate antecedent basis, as well as for reasons similar to the objections to claim 19 regarding 
the term "possible meaning" as originally claimed. In response, applicants have replaced the 
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above phrases with "a gesture mode", "a speech mode" and "meaning", respectively. Thus, 
claim 29 is now clear and posses adequate antecedent basis for all terms. 

Applicants request the removal of the foregoing objections to the claims. 

Rejection: 35 U.S.C. §101 

The Office Action rejected claims 1-53 under 35 U.S.C. §101 as being directed to 
nonstatutory subject matter. Applicants have claims 1, 2, 3 and 45. 

The Office Action rejects claims 1-53 as being abstract ideas without a claimed limitation 
to a practical application. Applicants disagree. Multimodal recognition (e.g., speech, gesture 
recognition) is a practical application of the claimed elements of claims 1-53 in and of itself. 
Applicant refers the Examiner to the "Examination Guidelines for Computer-Related Inventions, 
Final Version" issued by the Patent and Trademark Office published on February 28, 1996 in the 
Federal Register at 61 Fed. Reg. 7478 ("Guidelines") and as has been published in revised form 
as MPEP § 2106. Applicant particularly notes Section IV(B)(2)(e) of the Guidelines, which 
states 

"Office personnel must analyze the claim as a whole, in light of the specification, to 
understand what subject matter is being manipulated and how it is being manipulated. 
During this procedure, Office personnel must evaluate any statements of intended use or 
field of use, any data gathering step and any post-manipulation activity. Only when the 
claim is devoid of any limitation to a practical application in the technological arts 
should it be rejected under § 101." (emphasis added) 

Support for this standard is also provided in Markman v. Westview Instruments, in which the 
court held: 



Office personnel must rely on the applicant's disclosure to properly determine the 
meaning of the claims. Markman v. Westview Instruments, 52 F.3d 967, 980, 34 USPQ2d 
1321, 1330 (Fed. Cir.) (en banc), aff 'd, U.S. , 116S.Q. 1384(1996). 

Thus, a rejection under 35 U.S.C. §101 is proper only if, in light of the specification, a claim is 
devoid of any limitation to a practical application in the technological arts. Using this test, in 
light of the specification, the present invention as claimed is not devoid of any limitation to a 
practical application. 
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However, in order to expedite prosecution of this application, claims 1, 2 and 3 have been 
amended as suggested by the Examiner to add the limitation: "means for receiving said 
utterance" and claim 45 has been amended to add the limitation "receiving said multimodal 
utterance". 

Thus, for the foregoing reasons, claims 1, 2, 3 and 45 as amended are not directed to 
purely abstract ideas and posses adequate claimed structure and adequately identified practical 
application. Accordingly, claims 1, 2, 3 and 45 are directed to statutory subject matter. As a 
result, claims 4-44 and 46-53 are also directed to statutory subject matter. 

Applicants request the removal of the rejection under 35 U.S.C. §101. 

Rejection: 35 U.S.C. §102(b) 

The Office Action also rejected claims 1-53 under 35 U.S.C. §102(b) as being anticipated 
by the Sharma reference cited herein above. In order for a claim to be anticipated under 35 
U.S.C. 102, each and every limitation of the claim must be found either expressly or inherently 
in a single prior art reference. PIN/NIP. Inc. v. Platte Chem. Co. , 304 F.3d 1235, 1243 (Fed. Cir. 
2002). Applicants traverse. In the present case, Sharma does not show each and every limitation 
of claims 1-53. Therefore, Applicants request the withdrawal of the rejection under 35 U.S.C. 
102(b). 

Claim 1 : Regarding claim 1 , the Office Action cites FIG. 3 of Sharma as teaching the 
element of claim 1 of: "each finite state mode recognition system outputting at least one 
recognition lattice for each associated mode." The Office Action cites FIG. 3 and pages 856-858 
of Sharma as teaching the claim element of "an n-tape finite state device that inputs n-1 
recognition lattices from the plurality of finite-state mode recognition subsystems and outputs the 
multimodal meaning based on the n-1 recognition lattices." Applicants disagree that the cited 
portions of Sharma teach these elements of claim 1 . FIG. 3 does not show a lattice as 
contemplated by claim 1. Instead, FIG. 3 only shows the different types of input modes possible 
(e.g., typing, handwriting, body movement, hand pressure, etc) and maps those modes to 
categories of input (e.g., position/motion, audio, video, etc). The lattice as claimed in claim 1 is 
described at least at page 7, paragraph [0023] of the specification. As described in that passage: 

the term lattice denotes a directed and labeled graph, which is possibly weighted. 
IN each lattice, there is typically a designated tart node "s" and a designated final node 
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"t". Each possible pathway through the lattice from the start note s to the final node t 
induces a hypothesis based on the arc labels between each pair of nodes in the path. For 
example, in a word lattice, the arc labels are words and the various paths between the 
start node s and the final node t form sentences. 

Accordingly, the term lattice in claim 1 refers to a lattice in the actual utterance recognition 
process, not a mapping of a specific input mode to a category of input. The Sharma reference 
makes no reference, teaching or other suggestion of the lattice as contemplated by the present 
application and claimed in claim 1. Thus, this element is not taught by Sharma. 

Similarly, the cited portions of Sharma do not teach or make any other reference to an n- 
tape finite state device that inputs n-1 recognition lattices as taught by the respective claim 1 
element quoted above. Instead, pages 856-858 of Sharma discuss the different modalities (types 
of input) used in human-computer interface (HCI) and how those modalities are mapped to input 
categories for use in HCI applications. In other words, the cited passage of Sharma merely 
describes FIG. 3 in large part. However, no reference or teaching of an n-tape finite state device 
that inputs n-1 recognition lattices is made. Accordingly, this element is also not taught by 
Sharma. 

Thus, for the foregoing reasons, Sharma does not teach all the elements of claim 1 and, as 
a result, claim 1 is not anticipated by Sharma. Accordingly, claim 1 is allowable over Sharma. 

Claim 2: The Office Action next rejects claim 2 over the same cited passage of Sharma, 
specifically FIG. 3 and pages 856-858 of that reference. In particular, the Office Action states 
that this passage of Sharma teaches the following elements of claim 2: 

a pair of finite-state mode recognition systems, each finite-state mode recognition 
system usable to recognize one of the associated modes, each finite-state mode 
recognition system outputting at least one recognition lattice for each associated mode; 
and 

a multimodal recognition system that inputs a recognition lattice from each of the 
pair of mode recognition systems and outputs the multimodal meaning for the pair of 
associated modes based on the plurality of recognition results, comprising: 

a first system that inputs the pair of recognition lattices and outputs a 
combined recognition finite-state transducer; 

a second system the inputs the combined recognition finite-state 
transducer and outputs a combined recognition finite-state machine, and 

a third system that inputs the combined recognition finite-state machine 
and a multimodal meaning grammar and outputs the multimodal meaning. 
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However, for the reasons as described above in association with claim 1 , the cited portions of 
Sharma do not teach anything related to lattices as is contemplated and claimed in claim 2 of the 
present application. In addition, these portions do not teach outputting a combined recognition 
finite-state transducer, a combined recognition finite-state machine or outputting a multimodal 
meaning as a function of said finite state machine and a multimodal meaning grammar. Once 
again, FIG. 3 and pages 856-858 of Sharma only discuss the different modalities (types of input) 
used in human-computer interface (HCI) and how those modalities are mapped to input 
categories for use in HCI applications 

For these reasons, Sharma does not teach all the elements of claim 2. Therefore, claim 2 
is not anticipated by the Sharma reference. As a result, claim 2 is also allowable over the 
Sharma reference. 

Claim 3 : Regarding claim 3, the Office Action once again cites the same passage of 
Sharma, FIG. 3 and pages 856-858 as teaching the element of claim 3 of "wherein each of the 
plurality of mode recognition subsystems and the multimodal recognition subsystem includes at 
least one finite-state machine having at least one tape." However, at no point in the cited 
portions does Sharma teach this element. Instead, the cited portions of Sharma merely teach that 
which is described above. 

For these reasons, Sharma does not teach all the elements of claim 3. Therefore, claim 3 
is not anticipated by the Sharma reference. As a result, claim 3 is also allowable over the 
Sharma reference. It follows that claims 4-44 are allowable as being dependent upon an 
allowable base claim. 

Claim 4 : The Office Action next rejects claims 4, stating that pages 861-863 of the 
Sharma reference teach the claim element of: 

a first subsystem . . . that generates a first finite-state transducer that relates the 
input recognition results from each of the at least one mode recognition subsystems to a 
recognition model of at least one other mode recognition subsystem. 

However, while the cited portion of Sharma discusses how features from individual modalities 
can be integrated into more complex multimodal features, that portion of Sharma does not teach 
the above-cited claim element of claim 4 of a first subsystem . . . that generates a first finite state 
transducer . . Thus, claim 4 is allowable for this additional reason. 
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Other Dependent Claims : The Office Action next rejects claims 5-44, citing pages 864- 
866 of Sharma. This passage of Sharma discusses a case study of multimodal integration, at a 
relatively high level, as well as a review of various other prior multimodal systems and 
applications. Claims 5-7, 9-10, 15-21, 25-31 and 34-35 of the present application are rejected 
based on the above-cited passage of Sharma. These claims claim a finite state transducer that is 
input into a system or subsystem to produce in various ways a second finite state transducer, a 
projection of a finite state transducer, a finite state machine, or a lattice. At no point in the 
discussion of pages 864-866 of Sharma is there any reference or teaching of such a finite state 
transducer being input to a system or subsystem to produce the transducers/machine/lattice as 
claimed in these claims. Accordingly, Sharma does not anticipate claims 5-7, 9-10, 15-21, 25-31 
and 34-35 and, therefore, these claims are allowable for this additional reason. 

Claims 45-53 : Finally, the Office Action rejects claims 45-53 because those claims are 
"method claims to be implemented on the system claims 3-44, and are similar in scope and 
content and are rejected under similar rationale." Thus, since claims 3-44 are allowable as cited 
above, claims 45-53 are allowable according to similar rationale set forth above. 

Generality of the Rejections : 

The Office Action has cited generally multiple-page portions of Sharma as the basis of its 
rejection of the claims in the present application. 37 CFR 1.104(c)(2) states: 

In rejecting claims for want of novelty or for obviousness, the examiner must cite the best 
references at his or her command. When a reference is complex or shows or describes 
inventions other than that claimed by the applicant, the particular part relied on must be 
designated as nearly as practicable. The pertinence of each reference, if not apparent, 
must be clearly explained and each rejected claim specified. 

It is not apparent to Applicants which precise portions of the Sharma reference the Office intends 
to cite as teaching the precise claim elements of the pending claims. Accordingly, if the Office 
maintains the rejection of claims 1-53, Applicants respectfully request that these rejections be 
pointed out with greater particularity, as is required by 37 CFR 1.104(c)(2). 

Conclusion : 

Applicants have amended claims 2, 6, 7, 1 1, 12, 19 and 29 to correct the informalities 
identified by the Office. Claims 1, 2, 3 and 45 have been amended in response to the rejection 
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under 35 U.S.C. §101. No new matter has been introduced as a result of these amendments. 
The phrase "means for receiving said utterance" in claims 1, 2 and 3 and the phrase "receiving 
said multimodal utterance" in claim 45 are supported at least at paragraph [0024] of the present 
application. Applicants have traversed the rejections under 35 U.S.C. §102(b). 

For the foregoing reasons, claims 1-53, as amended, are within the technological arts and 
produce a useful, concrete, and tangible result. Therefore, claims 1-53 are directed to statutory 
subject matter under 35 U.S.C. §101. Also, for the above reasons, the cited portions of the 
Sharma reference do not anticipate all elements of claims 1-53 under 35 U.S.C. § 102(b). 
Applicants have requested that, should the Office maintain the objection to claim 16 or the 
rejection of claims 1-53 under 35 U.S.C. § 102(b), those objections/rejections be pointed out 
with the particularity required under 37 CFR 1.104(c)(2). 

Applicant requests allowance of all claims. 



Date: November 4, 2005 
AT&T Corp. 
Room 2A-207 
One AT&T Way 
Bedminster, NJ 07921 



Respectfully submitted, 




Tel.: 973-533-1616 
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